tudorizer/distilroberta-ai-job-embeddings

SentenceTransformer based on sentence-transformers/all-distilroberta-v1

This is a sentence-transformers model finetuned from sentence-transformers/all-distilroberta-v1 on the ai-job-embedding-finetuning dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-distilroberta-v1
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • ai-job-embedding-finetuning

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'RobertaModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("tudorizer/distilroberta-ai-job-embeddings")
# Run inference
queries = [
    "Here\u0027s a concise job search query:\n\nData Analyst (5+ years) - Power BI \u0026 Excel expertise required for remote construction data analysis role with competitive pay and benefits.\n\nThis query highlights the essential skills and requirements mentioned in the job description, while excluding generic terms like data science or software engineering.",
]
documents = [
    'Qualifications)\n\n 5+ years of data analytic, data validation, data manipulation experience Six Sigma yellow or green belt certification Strong Power BI skills Strong Excel skills\n\nHow To Stand Out (Preferred Qualifications)\n\n Six Sigma Black Belt certification\n\n#DataAnalysis #RemoteWork #CareerGrowth #CompetitivePay #Benefits\n\nAt Talentify, we prioritize candidate privacy and champion equal-opportunity employment. Central to our mission is our partnership with companies that share this commitment. We aim to foster a fair, transparent, and secure hiring environment for all. If you encounter any employer not adhering to these principles, please bring it to our attention immediately. Talentify is not the EOR (Employer of Record) for this position. Our role in this specific opportunity is to connect outstanding candidates with a top-tier employer.\n\nTalentify helps candidates around the world to discover and stay focused on the jobs they want until they can complete a full application in the hiring company career page/ATS.',
    'Skill set Required: Primary:Python, Scala, AWS servicesNoSQL storage databases such Cassandra and MongoDBApache Beam and Apache SparkAmazon Redshift, Google BigQuery, and Snowflake Secondary:Java, Go languageMicroservices frameworks such as Kubernetes and Terraform.',
    "experienced data scientist who thrives on innovation and craves the vibrancy of a startup environment.\nResponsibilitiesProven experience in applying advanced data science algorithms such as neural networks, SVM, random forests, gradient boosting machines, or deep learning.Demonstrable expertise in at least three classes of advanced algorithms.Prior experience with live recommender systems and their implementation.Proficiency in deep learning frameworks, preferably TensorFlow.Proven track record in implementing scalable, distributed, and highly available systems on Cloud Platform (AWS, Azure, or GCP).Strong machine learning and AI skills.Strong communication skills, adaptability, and a thirst for innovation.High autonomy, ownership, and leadership mentality are crucial as you will be a pivotal member shaping our organization's future.Strong skills in data processing with R, SQL, Python, and PySpark.\nNice to haveSolid understanding of the computational complexity involved in model training and inference, especially in the context of real-time and near real-time applications.Familiarity with the management and analysis of large-scale assets.A team player with a collaborative mindset who is eager to learn and apply new methods and tools.A sense of pride and ownership in your work, along with the ability to represent your team confidently to other departments.",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.4716, -0.1088, -0.0557]])

Evaluation

Metrics

Triplet

• Datasets: ai-job-validation and ai-job-test
• Evaluated with TripletEvaluator

Metric	ai-job-validation	ai-job-test
cosine_accuracy	0.9902	1.0

Training Details

Training Dataset

ai-job-embedding-finetuning

• Dataset: ai-job-embedding-finetuning at 5f56700
• Size: 816 training samples
• Columns: query, job_description_pos, and job_description_neg
• Approximate statistics based on the first 816 samples:

  	query	job_description_pos	job_description_neg
  type	string	string	string
  details	min: 33 tokens mean: 91.07 tokens max: 190 tokens	min: 7 tokens mean: 347.95 tokens max: 512 tokens	min: 7 tokens mean: 351.06 tokens max: 512 tokens

• Samples:

  query	job_description_pos	job_description_neg
  Here's a concise job search query with 3 specialized skills or areas of expertise that are distinct to the role: Data scientist positions GS-9 BLS Bureau of Labor Statistics Note: I excluded generic data science and software engineering skills, focusing on keywords related to the specific responsibilities and requirements listed in the job description.	Requirements Conditions of Employment Must be a U.S. Citizen.Must be at least 16 years old.Requires a probationary period if the requirement has not been met.Candidate required to obtain the necessary security/investigation level. Qualifications BASIC REQUIREMENTS: Degree: Mathematics, statistics, computer science, data science, or field directly related to the position. The degree must be in a major field of study (at least at the baccalaureate level) that is appropriate for the position. OR Combination of education and experience: Courses equivalent to a major field of study (30 semester hours) as shown in paragraph A above, plus additional education or appropriate experience. SPECIALIZED EXPERIENCE:In addition to the above basic requirements, applicants must have 52 weeks of specialized experience equivalent to at least the next lower grade level, GS-07, in the Federal Service. Specialized Experience is the experience that equipped the applicant with the particular know...	skills and supercharge careers. We help discover passion—the driving force that makes one smile and innovate, create, and make a difference every day. The Hexaware Advantage: Your Workplace BenefitsExcellent Health benefits with low-cost employee premium.Wide range of voluntary benefits such as Legal, Identity theft and Critical Care CoverageUnlimited training and upskilling opportunities through Udemy and Hexavarsity Experienced Data ScientistVery Strongly in AI and ML Primary Skills - Minimum 4 to 6 years of experience in AI/ML application build Experience in Generative AI with at least one major cloud LLM Experience in gathering requirements from the client Experience in designing the architecture Should have managed multiple PODs - Product Oriented Development Teams Experience in delivering back the application for continuous operation Manages the continuous usage of the application Exposure in Agile practices Secondary Skills - Certifications in Generative AI Certifications in Agi...
  Here's a concise job search query with 3 specialized skills: Business Analyst - Operations Reporting (SQL, Tableau, SDLC) This query highlights the key skills required for the role, such as SQL and Tableau expertise, experience with software development life cycles (SDLC), and business analysis capabilities.	skills, including prioritizing, problem-solving, and interpersonal relationship building.Strong experience in SDLC delivery, including waterfall, hybrid, and Agile methodologies.Experience delivering in an agile environment.Skills:Proficient in SQLTableau	requirements, ultimately driving significant value and fostering data-informed decision-making across the enterprise. Additional Information Job Site: Atlanta, GA40 hours/weekEligible for Employee Referral Program: $1500If offered employment must have legal right to work in U.S. You Must Have Qualified applicants must have a Master’s degree or foreign equivalent in Business Analytics, Data Science, Statistics, Applied Mathematics, or related field and five (5) years of IT experience. Full term of experience must include: data science, machine learning; commercial analytics; and implementing advanced analytical solutions in a business context. Must possess (2) two years of experience in the following: managing analytics projects and interfacing with internal / external project stakeholders; advanced programming skills in Python and SQL; big data technologies, including Hadoop and Spark; on-the-job experience developing, validating, and deploying a wide variety of machine learning a...
  Here's a concise job search query: Java Full Stack Lead Experienced developer wanted for long-term contract in Seattle, WA. Must have 14+ years of experience with JSP, Servlets, Spring Boot, and team leadership skills. I excluded generic data science or software engineering skills, and highlighted the unique requirements such as Java Full Stack experience, team leadership skills, and specific technologies like Spring Boot and JSP.	experience as a lead full stack Java developer with strong JSP and servlets and UI development along with some backend technologies experience Another primary skill is Team handling and responsible for Junior developer’s code reviews and onsite/offshore coordination experience is a must. Preferable local candidates Required skills: We need resources with Java, JSP, Servlets, JavaScript, jQuery, HTML, CSS, MSSQL, SOAP, MVC frameworks Spring or Struts, Spring Boot, and Restful web services. The position must have the following: Minimum of 14+ years of hands-on Java development experience.Strong experience on Application development & solution & Design.Strong experience in debugging and analytical skills.5 years of hands-on JavaScript experience.Extensive experience in delivering enterprise solutions with JSP, Servlets, Security and MVC.Strong experience with programming HTML/CSS technologiesGood understanding in XML, XSD, and XSLT.Strong experience in developing and consuming REST/...	skills and data science knowledge to create real-world impact. You’ll work closely with your clients to understand their questions and needs, and then dig into their data-rich environments to find the pieces of their information puzzle. You’ll develop algorithms and systems and use the right combination of tools and frameworks to turn sets of disparate data points into objective answers to help clients make informed decisions. Ultimately, you’ll provide a deep understanding of the data, what it all means, and how it can be used. Work with us as we use data science for good. Join us. The world can’t wait. You Have:  2+ years of experience as a Data ScientistExperience with scripting languages, including SQL, Python, or RKnowledge of operations research techniques, including probability and statistical methodsAbility to clean and build analytical data sets or pipelines from existing sources, when data engineering resources aren’t availableAbility to obtain a security clearance Bachel...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

ai-job-embedding-finetuning

• Dataset: ai-job-embedding-finetuning at 5f56700
• Size: 102 evaluation samples
• Columns: query, job_description_pos, and job_description_neg
• Approximate statistics based on the first 102 samples:

  	query	job_description_pos	job_description_neg
  type	string	string	string
  details	min: 55 tokens mean: 89.42 tokens max: 177 tokens	min: 15 tokens mean: 356.35 tokens max: 512 tokens	min: 14 tokens mean: 347.45 tokens max: 512 tokens

• Samples:

  query	job_description_pos	job_description_neg
  Here's a concise job search query with 3 specialized skills: Machine Learning Engineer remote job opportunities with Python, TensorFlow experience This query highlights the required Python and TensorFlow expertise, which are distinct from generic data science or software engineering skills.	requirements and deliver innovative solutionsPerform data cleaning, preprocessing, and feature engineering to improve model performanceOptimize and fine-tune machine learning models for scalability and efficiencyEvaluate and improve existing ML algorithms, frameworks, and toolkitsStay up-to-date with the latest trends and advancements in the field of machine learning RequirementsBachelor's degree in Computer Science, Engineering, or a related fieldStrong knowledge of machine learning algorithms and data modeling techniquesProficiency in Python and its associated libraries such as TensorFlow, PyTorch, or scikit-learnExperience with big data technologies such as Hadoop, Spark, or Apache KafkaFamiliarity with cloud computing platforms such as AWS or Google CloudExcellent problem-solving and analytical skillsStrong communication and collaboration abilitiesAbility to work effectively in a fast-paced and dynamic environment	skills.50% of the time candidate will need to manage and guide a team of developers and the other 50% of the time will be completing the technical work (hands on). Must have previous experience with this (i.e., technical lead)Code review person. Each spring. Coders will do developing then candidate will be reviewing code and auditing the code to ensure its meeting the standard (final eye)Migrating to a data warehouse. Required Skills:Informatica, IICS data pipeline development experienceCloud Datawarehouse (Snowflake preferred), on-prem to cloud migration experience.Ability to perform peer SIT testing with other Cloud Data EngineersDatabase - MS SQL Server, Snowflake Nice to have:Medium priority: Informatica PowerCenter (high priority)Analytical reporting - Tableau / Qlik Sense / SAS / R (migrating existing reports - mostly Tableau / moving from Qlik View to Qlik Sense)Kafka, KubernetesFinance, Lease / Loan or Automotive experience is a plus. Candidate can expect a panel interview with...
  Here's a concise job search query with 3 specialized skills or areas of expertise that are distinct to the role: Enterprise Data Engineer Microsoft Dynamics 365 ETL Processes This query focuses on the specific requirements mentioned in the job description, such as experience with Microsoft Dynamics 365 and ETL processes. It excludes generic data science or software engineering skills unless they are explicitly highlighted as unique or advanced. Other possible variations could be: * Data Integration Specialist Microsoft Dynamics 365 * Enterprise Data Engineer Azure Platform * ETL Process Expert Microsoft Dynamics 365 Note that these queries can be adjusted based on the specific job search needs and preferences.	requirements and building relationships.Drive risk-based data and integration decisions to minimize ERP implementation risks.Lead data extraction, transformation, and loading from legacy sources into Dynamics 365.Design, develop, and troubleshoot integrations with Dynamics 365 and other systems.Develop and maintain documentation for data processes and integration architecture.Enhance the enterprise data strategy in collaboration with leadership.Build and deploy scalable data pipelines and APIs to support evolving data needs.Drive data integrations for future acquisitions and ensure data integrity and governance.Collaborate with stakeholders to design and implement data models, dashboards, and reports. Qualifications for the Enterprise Data Engineer include: Proficiency in ETL processes and tools, preferably with experience in Microsoft Dynamics 365.Knowledge of Azure data platforms and tools like Power Automate, Azure Synapse, SQL database, Power BI, and more.Experience with REST-ba...	Qualifications: Bachelor’s degree or higher in Computer Science, Data Science, Engineering, Mathematics, Applied Statistics, or related field.8 years of experience in building data science and machine learning solutions using Python, Scala, Spark DataBricks, SQL, or similar technologies.Experience in text GenAI & LLM.Deep understanding of probability, statistics, machine learning, anomalies/outliers’ detection, and data correlation/feature analysis.Strong problem-solving skills and algorithm design capabilities.Proficiency in Python coding and familiarity with relevant ML packages. Mainz Brady Group is a technology staffing firm with offices in California, Oregon and Washington. We specialize in Information Technology and Engineering placements on a Contract, Contract-to-hire and Direct Hire basis. Mainz Brady Group is the recipient of multiple annual Excellence Awards from the Techserve Alliance, the leading association for IT and engineering staffing firms in the U.S. Mainz Brady...
  Here's a concise job search query: Data Scientist - Production Operations Houston, TX	1-year contract	Strong scripting/programming skills in Python, time series analysis experience (OSI PI, PI AF), and data visualization. This query highlights the key requirements mentioned in the job description, excluding generic data science or software engineering skills.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	ai-job-validation_cosine_accuracy	ai-job-test_cosine_accuracy
-1	-1	0.9902	1.0

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 5.0.0
• Transformers: 4.53.0
• PyTorch: 2.7.1+cu126
• Accelerate: 1.8.1
• Datasets: 3.6.0
• Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}